System and method for providing dynamic roll-back reservations in time

ABSTRACT

A systems, method and computer-readable media are disclosed for providing a dynamic roll-back reservation mask in a compute environment. The method of managing compute resources within a compute environment includes, based on an agreement between a compute resource provider and a customer, creating a roll-back reservation mask for compute resources which slides ahead of current time by a period of time. Within the roll-back reservation mask, the method specifies a subset of consumers and compute resource requests which can access compute resources associated with the roll-back reservation mask and, based on received data, the method dynamically modifies at least one of (1) the period of time the roll-back reservation mask slides ahead of current time and (2) the compute resources associated with the roll-back reservation mask.

PRIORITY APPLICATION

The present application is a continuation of U.S. patent applicationSer. No. 13/205,385, filed Aug. 8, 2011, which is a continuation of U.S.patent application Ser. No. 11/208,138, filed Aug. 19, 2005, now U.S.Pat. No. 7,996,455, issued Aug. 9, 2011, which is a continuation-in-partof PCT Application PCT/US 05/21427, filed on Jun. 17, 2005, the contentsof which are incorporated herein by reference in their entirety.

BACKGROUND

1. Technical Field

The present disclosure relates to reservations in a cluster or morespecifically to a system and method of providing dynamic roll-backreservations for compute resources.

2. Introduction

The present disclosure relates to a system and method of allocationresources in the context of a grid or cluster of computers. Gridcomputing can be defined as coordinated resource sharing and problemsolving in dynamic, multi-institutional collaborations. Many computingprojects require much more computational power and resources than asingle computer or single processor can provide. Networked computerswith peripheral resources such as printers, scanners, I/O devices,storage disks, scientific devices and instruments, etc., can need to becoordinated and utilized to complete a task or a job.

Grid/cluster resource management generally describes the process ofidentifying requirements, matching resources to applications, allocatingthose resources, and scheduling and monitoring compute resources overtime in order to run applications and workload as efficiently aspossible. Each project will utilize a different set of resources andthus is typically unique. In addition to the challenge of allocatingresources for a particular job, administrators also have difficultyobtaining a clear understanding of the resources available, the currentstatus of the compute environment and real-time competing needs ofvarious users. One aspect of this process is the ability to reserveresources for a job. A workload manager will seek to reserve a set ofresources to enable the compute environment to process a job at apromised quality of service. One example of workload management softwareis the various compute environment management software available fromCluster Resources, Inc., such as the Moab™ Workload Manager, Moab™Cluster Manager, the Moab™ Grid Suite and the Moab™ Cluster Suite.

General background information on clusters and grids can be found inseveral publications. See, e.g., Grid Resource Management, State of theArt and Future Trends, Jarek Nabrzyski, Jennifer M. Schopf, and JanWeglarz, Kluwer Academic Publishers, 2004; and Beowulf Cluster Computingwith Linux, edited by William Gropp, Ewing Lusk, and Thomas Sterling,Massachusetts Institute of Technology, 2003.

It is generally understood herein that the terms grid and cluster areinterchangeable in that there is no specific definition of either. Ingeneral, a grid will include one or more clusters as will be shown inFIG. 1A. Several general challenges exist when attempting to maximizeresources in a grid. First, there are typically multiple layers of gridand cluster schedulers. A grid 100 generally includes a group ofclusters or a group of networked computers. The definition of a grid isvery flexible and can mean a number of different configurations ofcomputers. The definition can depend on how a compute environment isadministered and controlled via local control (clusters) or globalcontrol/administration (grids). The introduction here is meant to begeneral given the variety of configurations that are possible.

A grid scheduler 102 communicates with one or more cluster schedulers104A, 104B and 104C. Each of these cluster schedulers communicates witha respective resource manager 106A, 106B or 106C. Each resource managercommunicates with a respective series of compute resources shown asnodes 108A, 108B, 108C in cluster 110, nodes 108D, 108E, 108F in cluster112 and nodes 108G, 108H, 108I in cluster 114.

Local schedulers (which can refer to either the cluster schedulers 104or the resource managers 106) are closer to the specific resources 108and do not allow grid schedulers 102 direct access to the resources.Examples of compute resources include data storage devices such as harddrives and computer processors. The grid level scheduler 102 typicallydoes not own or control the actual resources. Therefore, jobs aresubmitted from the high level grid-scheduler 102 to a local set ofresources with no more permissions that then user would have. Thisreduces efficiencies and can render the reservation process moredifficult. When jobs are submitted from a grid level scheduler 102,there is access information about the person, group or entity submittingthe job. For example, the identity of the person who submitted the jobcan have associated with it a group of restrictions but also guaranteesof service, such as a guarantee that 64 processors will be availablewithin 1 hour of a job submission.

The heterogeneous nature of the shared resources also causes a reductionin efficiency. Without dedicated access to a resource, the grid levelscheduler 102 is challenged with the high degree of variance andunpredictability in the capacity of the resources available for use.Most resources are shared among users and projects and each projectvaries from the other. The performance goals for projects differ. Gridresources are used to improve performance of an application but theresource owners and users have different performance goals: fromoptimizing the performance for a single application to getting the bestsystem throughput or minimizing response time. Local policies can alsoplay a role in performance.

Within a given cluster, there is only a concept of resource managementin space. An administrator can partition a cluster and identify a set ofresources to be dedicated to a particular purpose and another set ofresources can be dedicated to another purpose. In this regard, theresources are reserved in advance to processing the job. By beingconstrained in space, the nodes 108A, 108B, 108C, if they needmaintenance or for administrators to perform work or provisioning on thenodes, have to be taken out of the system, fragmented permanently orpartitioned permanently for special purposes or policies. If theadministrator wants to dedicate them to particular users, organizationsor groups, the prior art method of resource management in space causestoo much management overhead requiring a constant adjustment to theconfiguration of the cluster environment and also losses in efficiencywith the fragmentation associated with meeting particular policies.

Reservations of compute resources were introduced above. To manage thejobs submissions, a cluster scheduler will employ reservations to insurethat jobs will have the resources necessary for processing. FIG. 1Billustrates a cluster/node diagram for a cluster 124 with nodes 120.Time is along the X axis. An access control list (ACL) 114 to thecluster is static, meaning that the ACL is based on the credentials ofthe person, group, account, class or quality of service making therequest or job submission to the cluster. The ACL 114 determines whatjobs get assigned to the cluster 110 via a reservation 112 shown asspanning into two nodes of the cluster. Either the job can be allocatedto the cluster or it can't, and the decision is determined based on whosubmits the job at submission time. Further, in environments where thereare multiple clusters associated with a grid, and workload istransferred around the grid, there is a continual difficulty of managingrestrictions and guarantees associated with each entity that can submitjobs. Each cluster will have constant alterations made to users andgroups as well as modifications of the respective compute environment.Currently, there is no mechanism to insure that up-to-date identityinformation for a particular user where workload submitted by that usercan be transferred to an on-demand site or to a remote cluster from thesubmitter's local environment.

One deficiency with the prior approach is that there are situations inwhich organizations would like to make resources available but only insuch a way as to balance or meet certain performance goals.Particularly, groups can establish a constant expansion factor and makethat available to all users or they can make a certain subset of usersthat are key people in an organization and give them special serviceswhen their response time drops below a certain threshold. Given theprior art model, companies are unable to have the flexibility over theircluster resources.

To improve the management of cluster resources, what is needed in theart is an improved method for a scheduler, a cluster scheduler orcluster/grid workload management system to manage resources. Furtherwhat is needed is an improved method of managing reservations such thatthe user of the compute environment is more efficient while maintainingpolicies and agreed qualities of service.

SUMMARY

Additional features and advantages of the disclosure will be set forthin the description which follows, and in part will be obvious from thedescription, or can be learned by practice of the disclosure. Thefeatures and advantages of the disclosure can be realized and obtainedby means of the instruments and combinations particularly pointed out inthe appended claims. These and other features of the present disclosurewill become more fully apparent from the following description andappended claims, or can be learned by the practice of the disclosure asset forth herein.

The present disclosure addresses deficiencies in the prior art byimproving upon the use of reservations that enforce service levelagreements. A roll-back reservation was introduced in the parent casePCT Application PCT/US 05/21427, filed on Jun. 17, 2005. A dynamicaspect of the roll-back reservation was also introduced in that case.This application presents further details regarding the function andoperation of the dynamic roll-back reservation of compute resources. Theroll-back reservation according to the present disclosure can bedynamically modified either in time or in space for the purpose ofincreasing the efficiency of the use of the compute environment.

The disclosure relates to systems, methods and computer-readable mediafor providing a dynamic roll-back reservation mask in a computeenvironment. The method of managing compute resources within a computeenvironment includes, based on an agreement between a compute resourceprovider and a customer, creating a roll-back reservation mask forcompute resources which slides ahead of current time by a period oftime. Within the roll-back reservation mask, the method specifies asubset of consumers and compute resource requests which can accesscompute resources associated with the roll-back reservation mask and,based on received data, the method dynamically modifies at least one of(1) the period of time the roll-back reservation mask slides ahead ofcurrent time and (2) the compute resources associated with the roll-backreservation mask.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features of the disclosure can be obtained, a moreparticular description of the disclosure briefly described above will berendered by reference to specific embodiments thereof which areillustrated in the appended drawings. Understanding that these drawingsdepict only typical embodiments of the disclosure and are not thereforeto be considered to be limiting of its scope, the disclosure will bedescribed and explained with additional specificity and detail throughthe use of the accompanying drawings in which:

FIG. 1A illustrates generally a grid scheduler, cluster scheduler, andresource managers interacting with compute nodes;

FIG. 1B illustrates a job submitted to a resource set in a computingenvironment;

FIG. 2A illustrates a method of creating a reservation mask;

FIG. 2B illustrates a method of providing a roll-back reservation mask;

FIG. 2C illustrates a method of creating a dynamic roll-back reservationmask;

FIG. 3A illustrates a reservation mask;

FIG. 3B illustrates another aspect of the reservation mask;

FIG. 4 illustrates a dynamic roll-back reservation mask; and

FIG. 5 illustrates another aspect of a dynamic roll-back reservationmask.

DETAILED DESCRIPTION

Various embodiments of the disclosure are discussed in detail below.While specific implementations are discussed, it should be understoodthat this is done for illustration purposes only. A person skilled inthe relevant art will recognize that other components and configurationscan be used without parting from the spirit and scope of the disclosure.

The present disclosure relates to resource reservations in the contextof a cluster or grid environment or any other grouping of computedevices or compute nodes that can have similar administrative mechanismsto cluster and grid computing. The cluster can be operated by a hostingfacility, hosting center, a virtual hosting center, data center, grid,cluster, on-demand computer center and/or utility-based computingenvironments. As introduced above, the disclosure will provide detailsfor how a roll-back reservation can be dynamically modified either intime or space or both for the purpose of using received data about thecompute environment, the job, the requestor, or any other type of datathat can affect the compute environment to be utilized to dynamicallymodify the roll-back reservation to improve efficiency.

The system embodiment of the disclosure can include a computing devicethat is operating a module or software package, such as ClusterResources' Moab™ software, to perform the steps and functions describedherein. The computing device can include the known hardware componentssuch as a processor, memory such as RAM and/or ROM, a bus linking thevarious components, disk storage of any type, communication devices suchas modems or network cards to enable communication with other computingdevices in a cluster or a grid, and so forth. The particular hardwarecomponents combining to make a computing device are not necessarilylimited to any specific set but can include any known orfuture-developed configuration of hardware. Not each computing device ina cluster can have a display and a means for user input such as akeyboard and mouse. It is contemplated, however, that the system wouldinclude a graphical user interface to allow users and administrators tosubmit workload and manage the compute environment. The system aspect ofthe disclosure can include multiple computing devices. In fact, thesystem can include the cluster or grid itself inasmuch as the multiplenodes including a cluster or grid can be required to operate a softwaremodule or components of a workload management module in order topractice the principles of the disclosure.

It is also contemplated that the system can include features such asmulti-modal interfaces for ease of interaction and control of thecompute environment. For example, the graphical user interface canutilize natural language dialog, touch-screen input, motion detectioninput, gesture input, mouse input or a combination of these types ofinput to enable improved efficiency for users and administrators of thecompute environment.

As part of the system embodiment of the disclosure, a computer programcan include various modules written in a computer programming language,such as the C programming language or any suitable language. The moduleswould perform specific functions as stated in the method embodiment ofthe disclosure. The modules can operate on a single or multiplecomputing devices. As such, each module can be configured to perform theparticularly recited function discussed herein.

The present disclosure allows the ACL (access control list) for areservation to have a dynamic aspect instead of simply being based onwho the requester is. The ACL decision making process is based at leastin part on the current level of service or response time that is beingdelivered to the requester. To illustrate the operation of the ACL,assume that a user submits a job and that the ACL reports that the onlyjob that can access these resources are those that have a queue timethat currently exceeds two hours. If the job has sat in the queue fortwo hours it will then access the additional resources to prevent thequeue time for the user from increasing significantly beyond this timeframe. The decision to allocate these additional resources can be keyedoff of utilization of an expansion factor and other performance metricsof the job.

Whether or not an ACL is satisfied is typically determined by thescheduler 104A. However, there is no restriction in the principle of thedisclosure regarding where or on what node in the network the process ofmaking these allocation of resource decisions occurs. The scheduler 104Ais able to monitor all aspects of the request by looking at the currentjob inside the queue and how long it has sat there and what the responsetime target is and the scheduler itself determines whether allrequirements of the ACL are satisfied. If requirements are satisfied, itreleases the resources that are available to the job. A job that islocated in the queue and the scheduler communicating with the scheduler104A. If resources are allocated, the job is taken from the queue andinserted into the reservation in the cluster or compute environment.

An example benefit of this model is that it makes it significantlyeasier for a site to balance or provide guaranteed levels of service orconstant levels of service for key players or the general populace. Bysetting aside certain resources and only making them available to thejobs which threaten to violate their quality of service targets itincreases the probability of satisfying it.

The disclosure now continues to discuss reservations further. An advancereservation is the mechanism by which the present disclosure guaranteesthe availability of a set of resources at a particular time. With anadvanced reservation a site now has an ability to actually specify howthe scheduler should manage resources in both space and time. Everyreservation consists of three major components: a list of resources, atimeframe (a start and an end time during which it is active), and theACL. These elements are subject to a set of rules. The ACL acts as adoorway determining who or what can actually utilize the resources ofthe compute environment such as a cluster. It is the job of the clusterscheduler to make certain that the ACL is not violated during thereservation's lifetime (i.e., its timeframe) on the resources listed.The ACL governs access by the various users to the resources. The ACLdoes this by determining which of the jobs, various groups, accounts,jobs with special service levels, jobs with requests for specificresource types or attributes and many different aspects of requests canactually come in and utilize the resources. With the ability to say thatthese resources are reserved, the scheduler can then enforce trueguarantees and can enforce policies and enable dynamic administrativetasks to occur. The system greatly increases in efficiency because thereis no need to partition the resources as was previously necessary andthe administrative overhead is reduced in terms of staff time becausethings can be automated and scheduled ahead of time and reserved.

As an example of a reservation, a reservation can specify that node002is reserved for user John Doe on Friday. The scheduler will thus beconstrained to make certain that only John Doe's jobs can use node002 atany time on Friday. Advance reservation technology enables many featuresincluding backfill, deadline based scheduling, QOS support, and metascheduling.

There are several reservation concepts that will be introduced asaspects of the disclosure. These include dynamic reservations,co-allocating reservation resources of different types, reservationsthat self-optimize in time, reservations that self-optimization inspace, reservations rollbacks and reservation masks. The main focus ofthe present disclosure is the roll-back reservation mask and how it canbe dynamically modified.

Dynamic reservations are reservations that are able to be modified oncethey are created. Attributes of a reservation can change based on afeedback mechanism that adds intelligence as to ideal characteristics ofthe reservation and how it should be applied as the context of itsenvironment or an entities needs change. One example of a dynamicreservation is a reservation that provides for a guarantee of resourcesfor a project unless that project is not using the resources it has beengiven. A job associated with a reservation begins in a clusterenvironment. At a given portion of time into processing the job oncompute resources, the system receives compute resource usage feedbackrelative to the job. For example, a dynamic reservation policy can applywhich says that if the project does not use more than 25% of what it isguaranteed by the time that 50% of its time has expired, then, based onthe feedback, the system dynamically modifies the reservation ofresources to more closely match the job. In other words, the reservationdynamically adjusts itself to reserve X % fewer resources for thisproject, thus freeing up unused resource for others to use.

Another dynamic reservation can perform the following step: if usage ofresources provided by a reservation is above 90% with fewer than 10minutes left in the reservation then the reservation will attempt to add10% more time to the end of the reservation to help ensure the projectis able to complete. In summary, it is the ability for a reservation toreceive manual or automatic feedback to an existing reservation in orderto have it more accurately match any given needs, whether the needs areassociated with the submitting entity, the community of users,administrators, etc. The dynamic reservation improves the state of theart by allowing the ACL to the reservation to have a dynamic aspectinstead of simply being based on who the requestor is. The reservationcan be based on a current level of service or response time beingdelivered to the requestor.

As another example of a dynamic reservation, consider a user who submitsa job wherein the reservation needs an ACL that requires that the onlyjob that can access a set of resources is a job with a queue timecurrently exceeding two hours. If the job has sat in the queue for twohours it will then access the additional resources to prevent the queuetime for the user from increasing significantly beyond this time frame.An administrator can also key the dynamic reservation off ofutilization, off of an expansion factor and other performance metrics ofthe job.

The ACL and scheduler are able to monitor all aspects of the request bylooking at the current job inside the queue and how long it has satthere and what the response time target is. It is preferable, althoughnot required, that the scheduler itself determine whether allrequirements of the ACL are satisfied. If the requirements aresatisfied, the scheduler releases the resources that are available tothe job.

The benefits of this model is it makes it significantly easier for asite to balance or provide guaranteed levels of service or constantlevels of service for key players or the general populace. By settingaside certain resources and only making them available to the jobs whichthreaten to violate their quality of service targets it increases theprobability of satisfying it.

Another reservation type is a self optimizing reservation in time. Inmany cases, people will request resources and request that they beavailable at a particular time. For example, a person is doing ademonstration and it happens to be from 2:00 pm to 4:00 pm. In manyother cases, people will simply have a deadline or simply wantprocessing as early as possible. With a self-optimizing in timereservation, the scheduler is actually able to lock in a set ofresources for a particular request and then over time evaluate thecluster resources and determine if it can actually improve on it andimprove on the reservation in such a way as to guarantee that it doesnot lose the resources that it has already made available.

With self-optimizing reservations in time, a particular request can comein request resources that meet the following criteria but the requesterprefers resources that meet a more increasingly strict criteria. Thescheduler, in establishing the reservation, can satisfy the requiredcriteria but not necessarily satisfy all the preferred criteria. Overtime, the scheduler, once it has established a reservation that meetsthe minimum criteria, can continue to look at newly freed up resourcesand determine if it can, to a larger, satisfy the preferred resourceneeds as well. This self optimizing reservation technology is alsouseful to work around resource failures in the case of a reservationthat has already had reserved all the resources it needs and it has anode failure. It can actually continue to locate resources andreallocate resources that are still up and running and be able tosatisfy the time frame it originally promised by excluding the failednode and picking up a newly available compute node.

With the above concepts about reservations in mind, the reservation maskis next introduced. FIG. 2A illustrates the steps taken to provide areservation mask for compute resources. The method includes identifyinga need type and a group of available resources (202), creating areservation mask over the identified group of resources (204) and if arequest from a consumer matches the need type, then constraining thecreation of a reservation for the consumer to only use resources withinthe reservation mask (206). The reservation mask therefore has adifferent purpose from the reservation itself. The mask is apolicy-enforcing mechanism to manage and constrain reservations.Identifying a need type and a group of available resources can be basedon an administrative policy or some other criteria.

If a request from the consumer does not match the need type, then noconstraints are enforced for creating a reservation for the request fromthe consumer. Creating the reservation mask can also involve specifyingat least one timeframe during which the reservation mask enforcesconstraints, such as during business hours, eastern time. The time framecan also be one or more independent or periodic time frames. The methodcan also provide for specifying an access control list that constrainswhich consumers or resource requests can utilize resources within thereservation mask. The need type can refer to a particular use, a user, agroup of users, a job source, a type of job submission, personalreservation, grid reservation, cluster reservation and so forth.

A personal reservation, for example, can consist of a reservation thatdedicates resource access to a specific user or group of users. If thepersonal reservation provides access to resources to a group of users,then each reservation and reservation timeframe are determined by a userin the group of users that requests the respective reservation. A gridreservation is a reservation requested from outside an administrativegroup.

Another aspect of reservation relates to a roll-back reservation in timeor a roll-back reservation mask. FIG. 2B illustrates this methodembodiment of the present disclosure. The method of managing computeresources within a compute environment includes establishing a policy toprovide compute resources within a fixed time from the reception of arequest for a reservation (210), creating a roll-back reservation maskwhich slides ahead of current time by the fixed time (212) and receivinga request for a reservation (214). Upon receiving the request for areservation, the roll-back reservation mask insures that computeresources will be available for reservation within the fixed timeaccording to the policy. The policy can be established according to anagreement with a requestor of compute resources and the provider ormanager of the compute resources. An example policy would insure thatthe requestor of resources would be able to reserve and have at apredetermined quality of service, 100 nodes, 3 GB of memory and acertain bandwidth of communication within six hours of a request.

Within the roll-back reservation mask, the mask analyzes computeresources according to the policy to insure that compute resources canbe reserved by the requestor within the fixed period of time. An exampleof the request for a reservation is a consumption request, where a userdesires to process a submitted job using the compute resources. Afterreceiving the reservation request, the roll-back reservation maskreserves the appropriate compute resources according to the request andthe policy such that within the fixed amount of time, the requestor canhave a reservation established and have access to his or her reservedresources.

The roll-back reservation mask can also be self-optimizing. Given thatthere is sufficient time to analyze the request or reservation and thecompute resources, the reservation mask can analyze whether a level ofservice can be improved for the reservation request and if the level ofservice can be improved, then the mask cancels the reservation ofcompute resources and reserves a second group of compute resources. Themask or some other compute process can perform some of these steps. Thisself-optimization process of modifying or canceling and re-issuingreservations to improve performance of either the compute environment orthe quality of service delivered to the requestor can occur until apredetermined point. For example, assume the policy requires that therequestor have resources reserved and available for use within one hourof the request. If the requestor requests a reservation for three hoursinto the future, the roll-back reservation mask has two hours until thefixed guaranteed time to optimize the request. When the time comes wherethe request needs to be honored within one hour, one aspect of thedisclosure requires the reservation to be set and thus not “covered” bythe reservation mask. The reservation in this sense has slipped out fromunderneath the reservation mask. This is shown by the reservations 406in FIG. 4 and FIG. 5.

The roll-back reservation mask 402, 502 has a length preferably based onthe agreement. This can be, for example, a several months or it can beindefinite or of infinite length. Preferably, the length of the mask402, 502 is associated with how far into the future it analyzes computeresources and a height associated with a guaranteed throughput.

FIG. 2C illustrates the primary embodiment of the disclosure. Thisembodiment relates to the method of providing a dynamic roll-backreservation mask in a compute environment. The method of managingcompute resources within a compute environment includes, based on anagreement between a compute resource provider and a customer, creating aroll-back reservation mask for compute resources which slides ahead ofcurrent time by a period of time (220). Within the roll-back reservationmask, the method specifies a subset of consumers and compute resourcerequests which can access compute resources associated with theroll-back reservation mask (222) and, based on received data, the methoddynamically modifies at least one of (1) the period of time theroll-back reservation mask slides ahead of current time and (2) thecompute resources associated with the roll-back reservation mask (224).After some discussion of standing reservations and “sandboxes”, theapplication will address further details about the dynamic roll-backreservation.

FIG. 3A illustrates a standing reservation. In cluster 302, there arestanding reservations shown as 304A, 304B and 304C. These reservationsshow resources allocated and reserved on a periodic basis. These areconsuming reservations meaning that cluster resources will be consumedby the reservation.

A reservation mask, mentioned above, allows a compute site to create“sandboxes” in which other guarantees can be made. The most commonaspects of this reservation are for grid environments and personalreservation environments. In a grid environment, a remote entity will berequesting resources and will want to use these resources on anautonomous cluster for the autonomous cluster to participate. In manycases it will want to constrain when and where the entities can reserveor utilize resources. One way of doing that is via the reservation mask.

FIG. 3B illustrates the reservation mask shown as creating sandboxes306A, 306B, 306C in cluster 310 and allows the autonomous cluster tostate that only a specific subset of resources can be used by theseremote requesters during a specific subset of times. When a requesterasks for resources, the scheduler will only report and return resourcesavailable within this reservation, after which point the remote entitydesires it, he can actually make a consumption reservation and thatreservation is guaranteed to be within the reservation mask space. Theconsumption reservations 312A, 312B, 312C, 312D are shown within thereservation masks.

In cluster 310 the reservation masks operate differently from consumingreservations in that they are enabled to allow personal reservations tobe created within the space that is reserved. ACL's are independentinside of a sandbox reservation or a reservation mask in that you canalso exclude other requesters out of those spaces so they're dedicatedfor these particular users.

The benefits of this approach include preventing local job starvation,and providing a high level of control to the cluster manager in that heor she can determine exactly when, where, how much and who can use theseresources even though he doesn't necessarily know who the requesters areor the combination or quantity of resources they will request. Theadministrator can determine when, how and where requestors willparticipate in these grids. A valuable use is in the space of personalreservations which typically involves a local user given the authorityto reserve a block of resources for a rigid time frame. Again, with apersonal reservation mask, the requests are limited to only allowresource reservation within the mask time frame and mask resource set,providing again the administrator the ability to constrain exactly whenand exactly where and exactly how much of resources individual users canreserve for a rigid time frame. The individual user is not known aheadof time but it is known to the system, it is a standard local clusteruser.

The reservation masks 306A, 306B and 306C define periodic, personalreservation masks where other reservations in a cluster 310 can becreated, i.e., outside the defined boxes. These are provisioning orpolicy-based reservations in contrast to consuming reservations. In thisregard, the resources in this type of reservation are not specificallyallocated but the time and space defined by the reservation mask cannotbe reserved for other jobs. Reservation masks enable the system to beable to control the fact that resources are available for specificpurposes, during specific time frames. The time frames can be eithersingle time frames or repeating time frames to dedicate the resources tomeet project needs, policies, guarantees of service, administrativeneeds, demonstration needs, etc. This type of reservation insures thatreservations are managed and scheduled in time as well as space. Boxes308A, 308B, 308C and 308D represent non-personal reservation masks. Theyhave the freedom to be placed anywhere in cluster including overlappingsome or all of the reservation masks 306A, 306B, 306C. Overlapping isallowed when the personal reservation mask was setup with a global ACL.A global ACL is an ACL that anyone can use. It is wide open in the sensethat anyone can take advantage of the resources within that space. Toprevent the possibility of an overlap of a reservation mask by anon-personal reservation, the administrator can set an ACL to constrainit is so that only personal consumption reservations are inside. Thesepersonal consumption reservations are shown as boxes 312B, 312A, 312C,312D which are constrained to be within the personal reservation masks306A, 306B, 306C. The 308A, 308B, 308C and 308D reservations, ifallowed, can go anywhere within the cluster 310 including overlappingthe other personal reservation masks. The result is the creation of a“sandbox” where only personal reservations can go without in any wayconstraining the behavior of the scheduler to schedule other requests.

Returning to the discussion of the dynamic roll-back reservation mask,another view of the dynamic roll-back reservation mask 402 is shown inFIG. 4. This reservation mask 402 has particular application forenforcing policies or allowing support for service level guarantees inservice level agreements. A level of service guarantee allows a site,cluster or grid to guarantee that a particular consumer or organizationor type of credential is guaranteed a certain quantity of resourceswithin a certain amount of time 408. The standard way to provide thoseguarantees would be to dedicate a block of resources that satisfy theneeds and would be statically and rigidly partitioned so that no oneelse could access it. The request of that organization could not extendbeyond the bounds of the dedicated block.

A self optimizing reservation will only slide forward barring resourcefailure of the actual compute resources. It does this by, when it makesa query to determine what resources are available, as part of analgorithm, the reservation determines that it has availability to bothfree resources and the resources it already has reserved. In such a caseit then performs an analysis and looks at resources that were recentlyfreed by other workload and other reservations that completed early(which is actually quite common in a cluster environment) and if it canfind that it can improve the level of service delivered to the requestor it will actually create the new reservation and will remove the oldreservation and make other adjustments as needed. A self optimizingreservation therefore has the ability to improve any given attribute ofservice to the submitting entity, community of users, administrators,etc.

With the present disclosure regarding the dynamic reservation roll-back,an administrator can create a reservation mask 402 which enforces itspolicy and continues to float in time a certain distance 408 ahead ofthe current time. Typically, the rectangular area of the reservation hasa height that corresponds to guaranteed throughput (or relates to anycomputing environment parameter such as a number of nodes, storage thatis associated with the mask, bandwidth, etc.) when processing jobs andthe horizontal distance that corresponds to the length in time of thereservation. The reservation mask 402 can correspond to a certain amountof time according to a service level agreement, such as 3 or 4 monthsfor example. The reservation mask 402 can extend into infinity as wellif there is no defined ending time. The reservation mask 402 is aprovisioning reservation and maintains the time offset 408 to thecurrent time.

To illustrate the roll-back reservation, consider a service levelagreement with a company to have twenty resources available within onehour of the request for the resources and that they can make the requestanytime. The time offset 408 can then be set to one hour and the companywill never wait more than one hour to reserve and use up to twentyresources. The reservation mask 402 monitors the resources and when arequest is made for resources, consumption reservations 404 areallocated and left behind 406 as the roll-back reservation maskmaintains its offset. Those that are left behind are not “covered” bythe reservation 402 any longer.

An implementation with reservation rollback would allow a site to set upbasically a floating reservation mask that extends from one hour in thefuture until a time further in the future, such as 4 or 8 hours in thefuture, and continues to slide forward in time. The reservation mask 402will only allow jobs from this organization can drop down requests orreserve host resources underneath the reservation mask. As time movesforward, the reservation mask slides forward in time so it alwaysmaintains a constant distance in the future allowing these guarantees404 to be created and maintained 406 on the cluster.

The time offset 408 can be static or dynamic. A static offset 408 willmaintain a constant offset time, such as one hour into the future. Thestatic offset will likely be set by a service level agreement wherein acompany requests that the resources become available within an hour. Theoffset 408 can also by dynamic. There can be requests in the servicelevel agreement where under a given event or set of events, the offsetwould change wherein the reservation slides closer or farther away fromthe current time to provide a guarantee of resources within ½ (insteadof 1 hour) or 2 hours in the future. There are a variety of ways to varythe offset. One can be to simply cancel the current sliding reservationand create a new reservation at a different offset. Another way would beto maintain the current reservation but slide it closer or farther awayfrom the current time. The factors that adjust the dynamic nature of theoffset can be based on company requests, the nature and use of thecluster resources, the time the request is made, historical information,statistical information (e.g., 90% of the time workload from user number12 finishes at least 15 minutes late) and so forth. For example, if therequest for resources is made at midnight on a Friday night, perhapsinstead of the 1 hour availability of resources, the hosting centeranalyzes the cluster resources and the time of the request anddetermines that it can deliver the resources in ½ hour. The company canhave a flexible offset where, if the request is made during a block oftime such as between 3-4:30 pm (near the end of the work day), theoffset can be shortened so that the job can be processed sooner. Themodifications to the offset can be automatic based on a feedback loop ofinformation or can be adjustable by an administrator.

The dynamic aspect of the period of time in which the reservation maskslides ahead of the current time is discussed next. This aspect of thedisclosure provides some flexibility in how soon resources need to beavailable after a request for a reservation. For example, if the fixedtime offset 408 is three hours and a user submits a request for areservation on Friday at 3:00 pm, the soonest the resources would beguaranteed to be available to process a submitted job is 6:00 pm. Thatcan be beyond the time that the user desires to wait to submit a job. Adynamically modifiable period of time allows for some parameters thatcan move up the period of time in which the resources can be available.FIG. 2C provides the basic steps of providing a dynamic roll-backreservation.

The policy can be based on an agreement with a submitter of requests forreservations or a service level agreement. The period of time 408 and/orthe resources associated with the reservation mask 410 can bedynamically modifiable based on a number of factors, such as parameterswithin the policy, events related to the compute environment (a clusterenvironment or a grid environment), historical information such asprevious jobs submitted by the submitter, events related to a timeassociated with a job submission or the job submission itself, a requestby a consumer (for example, for resources within one hour where the settime off-set 408 is currently three hours), factors such as arrival ofnew consumers to the compute environment, new resources added to theenvironment or made available for use by the environment such as accessto an on-demand computing center, node failures, system maintenanceactions, backlog thresholds being hit, administrative action, failuresto connect to peer-to-peer services such as where the computeenvironment can no longer overflow to another on-demand site, reportedstatistics related to the compute environment, a desire to increaseprotection to guarantee SLA terms locally and/or events related billing.Access to the compute environment is allowed for authorized users. Ascan be seen, there can be a number of factors that can play a role in ananalysis of whether the period of time from which resources must beavailable after a request is received can be modified (increased ordecreased).

Examples of a peer-to-peer environment include a group of clusters, eachcluster in the group running workload manager/cluster scheduler softwaresuch as Cluster Resource's MOAB™. The workload managers on each clustercan communicate data with the other workload managers about its systemutilization and availability for overflow workload. These workloadmanagers can negotiate and optimize the overall global workloadmanagement by making available unused resources for other clusters toutilize. Thus, as a dynamic roll-back reservation mask is created, thetime offset 408 and/or the resources 410, as well as other parameters,can be modified based on peer-to-peer information, resource availabilitythrough a peer-to-peer environment, and so forth.

Examples of the resources that can be dynamically modified 410 include,but are not limited to, processor nodes, bandwidth resources, licenses,memory, disk storage, and generic resources or policy slots. Theworkload manager software that manages the compute environment canreserve instances of a generic resource or instances of a service. Forexample, the reservation mask can be associated with a reservation of100 instances of a particular service slot. The number in thereservation can be dynamically increased or decreased as set forthherein as an aspect of the disclosure. The point here is that a workloadmanager and/or a scheduler of a compute environment can refer to aparticular resource or particular service available within the computeenvironment in a generic sense in that it considers the ability tosimply reserve instances of the particular service. These genericreferences to resources or services can be dynamically modified as partof a reservation mask just as a physical resource such as 512 MB of harddisk space.

The following is an example of historical information that can trigger adynamic modification of the time offset 408 and/or the resources 410associated with a reservation mask. Assume that under a service levelagreement (SLA), a consumer is promised a 95% success rate in achievingfulfillment of SLA promises, such as resources available for processingjobs within 2 hours of a request. Historical information can trackwhether the success rate has always been fulfilled. For example, if theconsumer has a 100% success rate, wherein the SLA promises 95%, then atleast one of the time offset 408 or the resource amount 410 can beadjusted to enable others users easier access to the resources becausein so doing, the consumer can still “lose” some efficiency but stillmaintain the promised 95% success rate.

The time off-set 408 and resource amount 410 can be adjusted either way.If a consumer has been promised 95% success rate and historically onlyachieves 90%, then the time offset 408 and resource amount 410 can bedynamically adjusted to increase the success rate for that user. Anotherexample can be that compute environment utilization is dropping offsignificantly. In this case, the system can achieve a high quality ofservice success rate even with backing off SLA guarantees established bythe parameters 408 and 410. Therefore, these adjustments can modify theutilization and responsiveness for subsets of jobs within the computeenvironment.

Other features associated with the dynamic roll-back reservation maskare also contemplated. For example, the system can also receive data anddynamically modify, in addition to or as an alternate to the time offset408, the resources 410 associated with the roll-back reservation mask.

The parameters 408 and 410 can also be dynamically modified according toa negotiation between one entity and another. For example, when aconsumer submits a request or a job with a modification requestassociated with time 408 or resources 410, then the consumer essentiallyis requesting a negotiation with the workload manager (or module thatgoverns the access, reservation, and consumption) of the computeenvironment or the compute resource provider. This can, as mentionedabove, be a request for improved time, more resources for a certaincost, etc. As another example, the user can actually bid to pay more fora favorable parameter adjustment and the use of compute resources can be“auctioned” off by the provider. In this regard, the user offers to payan additional amount for a dynamic modification.

If a user has a prepaid amount in an account that is debited asresources are consumed by the user, a policy can be set that relaxes thequality of service if the consumer consumes the prepaid amount in theaccount. Where the consumer is operating “in the red”, the computeenvironment provider can drop the credentials or quality of service forthat user until the account is paid up. There also can be a graduatedscale where as the account gets further and further into the red, thequality of service continues to get worse and worse.

Another scenario is that a first hosting center can have a workloadmanager, such as Cluster Resource Inc.'s MOAB™ software, that negotiateswith a second hosting center's workload manager for access to and use ofcompute resources. This is an example of a peer-to-peer negotiation.Most negotiations take resources, pricing, consumer credentials, groupcredentials, resource availability, historical information, and so forthinto account in resolving conflicts and engaging in negotiations overwhether to and by how much should the system dynamically modify theseparameters.

Where a user negotiates with the provider over modifications of theparameters 408, 410, the negotiation is based on what the user isallowed to do. For example, the SLA or other policy will place limits onindividual users, such as they are only allowed 50 nodes at a time, andso forth. Any negotiation will have these policies as limits on thescope of the negotiation and when, if at all, those policies can beviolated. Where an administrator is requesting and negotiating over amodification, the negotiation is not likely as limited by policies andis more about what the system is capable of delivering in the modifiedenvironment. The peer-to-peer negotiation introduced above between oneworkload manager and another workload manager can also have policyconstraints based on the individual compute environment, users and/orgroups within the respective environments and so forth.

The dynamic modification of parameters 408 and 410 can also be relatedto at least one of a policy and a feedback loop. For example, the policycan be embedded in a SLA or can be associated with the computeenvironment. The policy can relate to actions to take based on how wellthe SLA is being fulfilled or how well non-SLA metrics are beingfulfilled such as overall system utilization. The feedback loop aspectof the disclosure relates to each reservation mask having a purpose. Howwell the mask fulfils its purpose is fed back into the system toconsider thresholds and whether dynamic modifications need to be made.For example, a hosting center can have a 95% success rate goal. If, overtime, the hosting center can reduce the amount of resources reserved forusers or groups, and still match its 95% goal, then the hosting centercan dynamically adjust downward the services to one or more clients andstill meet that client's SLA requirements while increasing resources toother users and groups while still maintaining its hosting center policygoal.

The reservation rollback policy mask is stackable, allowing multipledifferent types of service or service level agreements to besimultaneously satisfied and share a collection of resources. Thisfeature is illustrated in FIG. 5. A reservation 502 is shown and cangenerally be considered as an aggregation of requests from various masks504, 506, 508 510. These are aggregated into one space 502 which willthen allow reservations to be created on a first come first serve basis,or based on other factors. If these reservation masks 504, 506, 508 and510 are stacked with individual offsets from the current time (notshown), the administrator can allow the masks to be partitioned amongconsumers. A useful component of this stackable approach is thecapability to have an enveloping reservation 502 created with a totalquantity of resource and rollback time offset 408 and a duration to theend of the SLA. Once that reservation space is established or paid for,as a service, the hosting center sub-partitions the space usingreservation to provide service guarantees, response time guarantees,quantity or resources guarantees taking advantage of the stackingcapability.

A company can therefore establish the enveloping reservation 502 andrequest from the hosting center that they partition the space accordingto various organizations within the enveloping reservation 502. Thiseliminates the need for a large entity to have its own group of clustersof computer. The parameters associated with the time offset 408 andresources allocation 410 can be dynamically modifiable in this exampleas well based on the factors discussed above. In this case, additionalfactors can be considered when dynamically modifying these parametersthat relate to the stacked service or service level agreements thatshare the collection of resources.

Embodiments within the scope of the present disclosure can also includecomputer-readable media for carrying or having computer-executableinstructions or data structures stored thereon. Such computer-readablemedia can be any available media that can be accessed by a generalpurpose or special purpose computer. By way of example, and notlimitation, such computer-readable media can include RAM, ROM, EEPROM,CD-ROM or other optical disk storage, magnetic disk storage or othermagnetic storage devices, or any other medium which can be used to carryor store desired program code means in the form of computer-executableinstructions or data structures. When information is transferred orprovided over a network or another communications connection (eitherhardwired, wireless, or combination thereof) to a computer, the computerproperly views the connection as a computer-readable medium. Thus, anysuch connection is properly termed a computer-readable medium.Combinations of the above should also be included within the scope ofthe computer-readable media.

Computer-executable instructions include, for example, instructions anddata which cause a general purpose computer, special purpose computer,or special purpose processing device to perform a certain function orgroup of functions. Computer-executable instructions also includeprogram modules that are executed by computers in stand-alone or networkenvironments. Generally, program modules include routines, programs,objects, components, and data structures, etc. that perform particulartasks or implement particular abstract data types. Computer-executableinstructions, associated data structures, and program modules representexamples of the program code means for executing steps of the methodsdisclosed herein. The particular sequence of such executableinstructions or associated data structures represents examples ofcorresponding acts for implementing the functions described in suchsteps.

Those of skill in the art will appreciate that other embodiments of thedisclosure can be practiced in network computing environments with manytypes of computer system configurations, including personal computers,hand-held devices, multi-processor systems, microprocessor-based orprogrammable consumer electronics, network PCs, minicomputers, mainframecomputers, and the like. Embodiments can also be practiced indistributed computing environments where tasks are performed by localand remote processing devices that are linked (either by hardwiredlinks, wireless links, or by a combination thereof) through acommunications network. In a distributed computing environment, programmodules can be located in both local and remote memory storage devices.

Although the above description may contain specific details, they shouldnot be construed as limiting the claims in any way. For example, otherparameters besides the time-off set 408 and resources 410 in thereservation mask can also be modified dynamically. For example, whetherthe reservation mask extends to infinity or not can be modified if itimproves a success rate of a compute environment. Other configurationsof the described embodiments of the disclosure are part of the scope ofthis disclosure. Accordingly, the appended claims and their legalequivalents should only define the disclosure, rather than any specificexamples given.

I claim:
 1. A method comprising: identifying compute resources within acompute environment to yield identified data; based on the identifieddata, creating a policy-enforcing mechanism that manages and constrainsestablishment of reservations of compute resources for use by workload;and establishing a reservation for compute resources within the computeenvironment, wherein the establishing of the reservation is eitherconstrained by the policy-enforcing mechanism or not constrained by thepolicy enforcing mechanism based on whether a request matches aparameter.
 2. The method of claim 1, wherein the parameter is a needtype.
 3. The method of claim 1, further comprising: creating a set ofnon-consumption reservation masks covering multiple nodes in the computeenvironment, wherein multiple reservations created from multipleconsumer requests are each constrained to only use at least a portion ofthe multiple nodes covered by the set of non-consumption reservationmasks.
 4. The method of claim 1, wherein the policy-enforcing mechanismis a non-consumption reservation mask.
 5. The method of claim 1, whereincreating the policy-enforcing mechanism further comprises specifying atleast one timeframe during which the policy-enforcing mechanism enforcesconstraints.
 6. The method of claim 5, wherein the at least one timeframe further comprises a plurality of independent time frames.
 7. Themethod of claim 1, further comprising: specifying an access control listthat constrains which consumers can utilize compute resources under thepolicy-enforcing mechanism.
 8. The method of claim 2, wherein the needtype comprises at least one of: a particular use, a user, a group ofusers, a workload source and a type of workload submission.
 9. Themethod of claim 2, wherein the need type is a personal reservation thatcomprises a reservation that dedicates resource access to at least oneof a user and a group of users.
 10. The method of claim 9, wherein ifthe personal reservation provides access to resources to a group ofusers, then each reservation and reservation timeframe are determined bya user in the group of users that requests the respective reservation.11. The method of claim 1, further comprising: modifying the reservationaccording to received data.
 12. The method of claim 11, wherein thereceived data is at least one of resource usage, system performance, apolicy and a criterion associated with the request.
 13. The method ofclaim 11, wherein modifying the reservation is bounded by a minimumthreshold and a maximum threshold.
 14. The method of claim 11, whereinmodifying the reservation further comprises modifying at least one of:an access control list, reserved resources and a time frame covered. 15.A computer-readable storage device that stores instructions which, whenexecuted by a computing device to manage compute resources in amulti-node compute environment, cause the computing device to performoperations comprising: identifying compute resources within a computeenvironment to yield identified data; based on the identified data,creating a policy-enforcing mechanism that manages and constrainsestablishment of reservations of compute resources for use by workload;and establishing a reservation for compute resources within the computeenvironment, wherein the establishing of the reservation is eitherconstrained by the policy-enforcing mechanism or not constrained by thepolicy enforcing mechanism based on whether a request matches aparameter.
 16. A system for managing compute resources within amulti-node compute environment, the system comprising: a processor; anda non-transitory computer-readable medium storing instructions which,when executed by the processor, cause the processor to performoperations comprising: identifying compute resources within a computeenvironment to yield identified data; based on the identified data,creating a policy-enforcing mechanism that manages and constrainsestablishment of reservations of compute resources for use by workload;and establishing a reservation for compute resources within the computeenvironment, wherein the establishing of the reservation is eitherconstrained by the policy-enforcing mechanism or not constrained by thepolicy enforcing mechanism based on whether a request matches aparameter.